cloud

Build an OpenSearch Cost Agent: Find Oversharded Indices, Missing ISM Policies, and Data Nobody Searches, Then Ship Rollover, Retention, and UltraWarm Fixes as PRs

Build an OpenSearch cost agent: find oversharded indices, unmanaged retention, and unsearched data, then ship ISM, template, and UltraWarm fixes as PRs.

November 21, 2026·12 min read·
#ai#sre#devops#aws#automation#cost-optimization#finops#observability

What This Agent Does

An OpenSearch cost agent reads _cat/indices, _cat/shards, per-index search and indexing counters, and Index State Management (ISM) status from an Amazon OpenSearch Service domain. It groups indices into patterns, then reports four kinds of waste per pattern: too many small shards, indices with no lifecycle policy, replicas kept on data that is never written again, and hot-tier storage spent on indices nobody queries. Its output is a pull request against the Terraform that owns the index templates and ISM policies. It holds no permission to search a document, delete an index, or change the domain.

The domain bill is instance hours plus EBS, and both are driven by things the bill does not show. Shard count sets how much JVM heap the cluster needs, which sets the instance size. Retention and replica count set the storage. Neither appears in Cost Explorer, which is why the domain keeps getting scaled up when the disk alarm fires, and nobody asks why a logging cluster is holding 400 days of debug output.

Step 1: Collect Shape, Not Content

Four read-only calls describe the whole domain:

# one row per index: shard layout, size, age
GET _cat/indices?format=json&bytes=b&h=index,pri,rep,docs.count,docs.deleted,pri.store.size,store.size,creation.date

# one row per shard copy: where the bytes sit
GET _cat/shards?format=json&bytes=b&h=index,shard,prirep,state,store,node

# cumulative counters per index
GET _all/_stats/search,indexing?level=indices

# which indices ISM manages, and with which policy
GET _plugins/_ism/explain/*

The agent's role is built from the cluster_monitor and indices_monitor action groups plus the ISM read actions. It deliberately lacks read, so a _search from the agent's credential is rejected by fine-grained access control. Everything it knows about an index is its name, its size, and its counters.

The counters need the same treatment as any cumulative statistic. search.query_total and indexing.index_total are kept per shard and start again from zero when a shard relocates or a node restarts, which on a managed domain happens during every blue/green deployment. So the agent stores a daily snapshot and works from differences:

# oscost/usage.py
def delta(prev: dict, cur: dict) -> dict:
    """prev/cur: {index: {"query_total": int, "index_total": int}}"""
    out = {}
    for name, c in cur.items():
        p = prev.get(name)
        if p is None or c["query_total"] < p["query_total"] \
                     or c["index_total"] < p["index_total"]:
            out[name] = {"reset": True, "queries": 0, "writes": 0}
            continue
        out[name] = {"reset": False,
                     "queries": c["query_total"] - p["query_total"],
                     "writes":  c["index_total"] - p["index_total"]}
    return out

A day flagged reset contributes nothing and restarts that index's evidence window. A claim such as "no queries for 35 days" requires 35 consecutive clean days in the agent's own table, not a number the cluster reported this morning.

Step 2: Group Into Patterns

Nobody fixes one index. The unit of work is the index template, so the agent strips date and sequence suffixes (app-logs-2026.11.20, orders-000042) and aggregates:

patternindicesprimaries eachreplicasavg primary shardtotal shardsISM policy
app-logs-*90510.8 GB900none
audit-*412122.1 GB1236none
orders-*63124 GB36orders-rollover

That table is already most of the finding. The model's job later is to explain it and write the fix, not to discover it.

Step 3: Shard Math Is Deterministic

AWS's sizing guidance is 10 to 30 GB per shard where search latency matters and 30 to 50 GB for write-heavy workloads such as logs. Its long-standing ceiling is 25 shards per GiB of JVM heap, and the service enforces a per-node shard limit on top (1,000 per node by default on most versions; check the quota page for yours). A shard costs heap and cluster-state size whether it holds 40 GB or 40 KB.

app-logs-* writes 4 GB of primary data a day into a daily index with five primaries. Each shard is 0.8 GB. Ninety days of retention with one replica is 900 shards. On three data nodes with 16 GiB of heap each, the guidance allows 1,200 shards for the whole domain, and one pattern is using three quarters of it.

# oscost/shards.py
import math

GIB = 2**30

def plan(daily_primary_bytes: int, retention_days: int, replicas: int,
         target_shard_gb: int = 30, max_index_age_days: int = 7) -> dict:
    # bytes one index receives before the age condition rolls it over
    per_index = daily_primary_bytes * max_index_age_days
    primaries = max(1, math.ceil(per_index / (target_shard_gb * GIB)))
    indices = math.ceil(retention_days / max_index_age_days)
    return {"primaries": primaries,
            "rollover": {"min_primary_shard_size": f"{target_shard_gb}gb",
                         "min_index_age": f"{max_index_age_days}d"},
            "total_shards": indices * primaries * (1 + replicas)}

For the example it returns one primary, rollover at 30 GB or seven days, and 26 shards in steady state. Going from 900 to 26 changes no storage, and that should be said plainly in the PR. What it changes is heap pressure, cluster-state size, and recovery time, which is what lets the domain drop an instance size later. That second step is a separate, human decision, in the same spirit as the EC2 right-sizing agent: prove the headroom first, resize afterwards.

Two more deterministic checks run here. Indices with zero documents that still hold shards are listed as a group, since they are usually a rollover-by-age policy firing on a stream that stopped. And indices where docs.deleted exceeds a third of docs.count and the write delta has been zero for a week are force-merge candidates, since the deleted documents still occupy disk until segments merge.

Step 4: Retention, Replicas, and Tier

No ISM policy. Any time-series pattern whose indices show policy_id: null in the explain output has no retention at all. It grows until someone deletes by hand. The agent computes bytes by age bucket so the PR can say what each retention choice would free.

Replicas on frozen data. An index with zero writes for 30 days and two replicas is paying for three copies of something that will never change. Dropping to one replica frees a third of its storage. The agent never proposes zero replicas on the hot tier, and it skips this check entirely on domains using Multi-AZ with Standby, where the replica count is part of the availability contract.

Hot storage for cold data. For indices with no writes and a query delta near zero, UltraWarm is the bigger lever: storage is S3-backed, billed on primary size only, so replicas stop costing anything. The wrapper checks the preconditions before the model ever sees the option: the domain has dedicated master nodes, warm nodes are enabled, and the index is no longer a write target. Warm indices are read-only, so anything that receives late-arriving documents is excluded.

Who else reads it. Query counters miss the consumers that run rarely. The agent lists alerting monitors through _plugins/_alerting/monitors/_search and Dashboards index patterns through the saved objects API (api/saved_objects/_find?type=index-pattern), and matches both against each pattern. A monitor that queries audit-* once a day is a consumer even if its query count is small.

Step 5: The Model Classifies a Pattern and Writes the Policy

The model gets one pattern: the aggregate row, the shard plan from Step 3, age buckets with bytes, query and write deltas per bucket, the monitors and index patterns that reference it, and the current template and policy if any.

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "actions": {"type": "array", "items": {"type": "string", "enum": [
            "fix_template_shards", "add_rollover_policy", "add_retention",
            "reduce_replicas_when_frozen", "warm_migration", "force_merge",
            "keep", "needs_human"]}},
        "retention_days": {"type": ["integer", "null"]},
        "evidence": {"type": "string"},
        "what_breaks_if_wrong": {"type": "string"},
    },
    "required": ["actions", "evidence", "what_breaks_if_wrong"],
}

SYSTEM = (
    "You review one OpenSearch index pattern using only the numbers provided.\n"
    "Propose retention only as a question: state how many bytes each of 30, 90, "
    "and 365 days would free and set retention_days to null unless the pattern "
    "carries a retention tag from the service catalog.\n"
    "needs_human: the name suggests audit, security, billing, or compliance "
    "data; a monitor or dashboard references indices older than the proposed "
    "warm or delete age; or the pattern is not time-series.\n"
    "Never propose shrinking or reindexing existing indices when retention is "
    "under 90 days; fixing the template and letting old indices expire is safer.\n"
    "Cite the figures given. Do not estimate dollar savings."
)

The retention rule is the one that matters. How long to keep logs is a legal and operational decision, and no counter can make it. The model may show what each option frees; it may not pick one.

For accepted actions the PR contains two Terraform resources from the opensearch-project/opensearch provider: an opensearch_composable_index_template with the corrected number_of_shards and rollover alias, and an opensearch_ism_policy whose body looks like this:

{
  "policy": {
    "description": "app-logs: rollover, drop replica when frozen, warm, delete",
    "default_state": "hot",
    "ism_template": [{ "index_patterns": ["app-logs-*"], "priority": 100 }],
    "states": [
      { "name": "hot",
        "actions": [{ "rollover": { "min_primary_shard_size": "30gb", "min_index_age": "7d" } }],
        "transitions": [{ "state_name": "warm", "conditions": { "min_index_age": "14d" } }] },
      { "name": "warm",
        "actions": [{ "warm_migration": {} }],
        "transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "90d" } }] },
      { "name": "delete", "actions": [{ "delete": {} }] }
    ]
  }
}

The Trap in That Policy

An ism_template applies to indices created after the policy exists. Existing indices are untouched until someone attaches the policy with POST _plugins/_ism/add/app-logs-*. And the moment that happens, ISM evaluates min_index_age against each index's creation date. Attach a 90-day delete policy to a pattern with 400 days of history and roughly 310 indices walk to the delete state within a few job intervals. There is no confirmation prompt and no trash folder.

So the wrapper computes, for every proposed policy, the exact list of existing indices that would be deleted or migrated on first evaluation, with bytes, and prints it at the top of the PR. It also states what the snapshots cover. The automated snapshots on a managed domain live in the cs-automated repository and are kept for 14 days. They are a recovery tool, not an archive, and the PR says so in those words when no manual snapshot repository holds the indices about to go.

Guardrails That Live in Code

  • Monitor-only role. No read, no indices:admin/delete, no ISM write actions. The agent cannot run the migration it recommends.
  • The only write is a pull request, following the PRs-not-kubectl pattern. The policy lands through terraform apply in the normal pipeline.
  • Attaching a policy to existing indices is never in the PR. Merging affects new indices only. The backfill command is printed for a human to run, with the first-evaluation deletion list beside it, behind the same approval gate as any destructive change.
  • Name-based exclusions are enforced by the wrapper. System indices (leading dot), and anything matching the configured compliance patterns, are removed from retention and warm proposals regardless of what the model returns.
  • Domain changes use the dry run. If a later step proposes an instance or volume change, the agent attaches the output of aws opensearch update-domain-config --dry-run, which reports whether the change triggers a blue/green deployment. It does not submit the change.
  • One pattern per PR. After merge the agent watches JVM memory pressure, search latency, and rejected-thread-pool counts for a week before proposing the next.

Run it in shadow mode first. The evidence window needs five weeks of snapshots anyway, and the weekly report alone tends to settle the retention argument once the owning team sees the age buckets.

Honest Limits

Query counters measure that a search touched an index, not that anyone needed the result. A Dashboards panel on app-logs-* with a wide time range touches every index in the pattern, so a nonzero count on a year-old index may be one person's forgotten default time picker. The reverse failure is worse: the incident investigation that needs last quarter's logs happens twice a year and will not appear in any 35-day window. Zero queries is evidence about the window only.

Fixing a template does nothing for indices that already exist. For short retention that is fine, since the oversharded indices age out. For patterns kept a year or more, the cleanup is _shrink or _reindex, both of which need free disk, a write block, and for shrink a copy of every shard on one node. The agent describes that work and leaves it to a person.

Fewer, larger shards are not free either. A 50 GB shard takes longer to relocate and recover than ten 5 GB shards, and a single primary caps indexing parallelism for that index. The 30 GB target is a starting point to be checked against indexing latency after the change.

UltraWarm trades cost for latency on the first query against uncached data, and migration itself runs a force merge that takes time and I/O. It also adds warm node hours, so it only pays above a certain volume. The agent reports bytes moved, not money saved, because that crossover depends on pricing it does not have.

Finally, the largest saving is often upstream. If half of app-logs-* is debug lines, the fix belongs in the shipper, the same way the CloudWatch Logs cost agent treats ingestion and the Prometheus cardinality agent treats unused series: data you never store is cheaper than data you tier well.

#ai#sre#devops#aws#automation#cost-optimization#finops#observability
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →