Build a Redis Memory Agent: Find Big Keys, No-TTL Keyspaces, and Eviction Storms on ElastiCache, Then Ship TTL and maxmemory-policy Fixes Without a FLUSHALL

What This Agent Does

A Redis memory agent samples the keyspace from a read replica, groups keys by pattern, and tells you which three or four prefixes own the memory, which of them never expire, which single keys are large enough to stall the event loop, and whether the evictions on your dashboard are a sizing problem or a policy problem. It then proposes fixes in the only form it is allowed to: a pull request to the owning service adding a TTL, a Terraform PR changing the parameter group, or a right-sizing recommendation with the cost delta attached. It never runs KEYS, never touches the primary, never deletes anything, and the one write it can perform, a rate-limited EXPIRE ... NX backfill, runs only after a human approves it.

The reason to build this is that Redis memory problems hide behind one number. DatabaseMemoryUsagePercentage at 91% tells you nothing about whether the 91% is a session cache that should have had a 30-minute TTL since 2023, a single leaderboard:global sorted set with four million members, or a legitimately full working set that needs a bigger node. Each of those has a different fix, two of them are free, and the one everyone reaches for first, the larger node, is the only one that costs money every month. Sorting it by hand means an engineer running SCAN in a terminal at 2 a.m., and that is toil with a blast radius.

Step 1: Sample Keys from a Replica, Never the Primary

The sampler connects to the reader endpoint, walks the keyspace with SCAN, and for every sampled key pipelines four cheap reads: type, remaining TTL, an estimated size, and how long since the key was last accessed. It stops at a budget, sleeps between batches, and aborts if the replica falls behind.

# redismem/sample.py
import re, time, redis

NOISE = [
    (re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"), "{uuid}"),
    (re.compile(r"\b[0-9a-f]{16,}\b"), "{hex}"),
    (re.compile(r"\d+"), "{n}"),
]

def pattern(key: str) -> str:
    for rx, repl in NOISE:
        key = rx.sub(repl, key)
    return key

def sample(reader_host: str, budget: int = 200_000, lfu: bool = False) -> list[dict]:
    r = redis.Redis(host=reader_host, port=6379, ssl=True, decode_responses=True)
    cursor, out = 0, []
    while len(out) < budget:
        cursor, keys = r.scan(cursor=cursor, count=1000)
        p = r.pipeline(transaction=False)
        for k in keys:
            p.type(k); p.pttl(k); p.memory_usage(k, samples=5)
            p.object("freq" if lfu else "idletime", k)
        res = p.execute(raise_on_error=False)
        for i, k in enumerate(keys):
            t, pttl, mem, idle = res[i*4:(i+1)*4]
            if isinstance(mem, Exception):            # key expired mid-pipeline
                continue
            out.append({"key": k, "pattern": pattern(k), "type": t,
                        "pttl_ms": pttl, "bytes": mem, "idle_s": idle})
        if cursor == 0:
            break
        info = r.info("replication")
        if info.get("master_link_status") != "up" or int(info.get("master_last_io_seconds_ago", 0)) > 5:
            raise RuntimeError("replica lagging; stop sampling")
        time.sleep(0.05)
    return out

Three details carry the safety story. SCAN with COUNT 1000 is a bounded amount of work per call, where KEYS * on a 40-million-key database blocks the server for seconds. MEMORY USAGE with SAMPLES 5 estimates a hash by sampling five fields rather than walking all of them, so a million-field hash costs the same as a ten-field one. And the replication check between batches is the kill switch: a replica that stops acknowledging its primary is a replica you are hurting.

The lfu flag matters more than it looks. OBJECT IDLETIME only works when the eviction policy is LRU-based. With allkeys-lfu or volatile-lfu, Redis returns an error instead of a number, and the agent has to read OBJECT FREQ, a logarithmic access counter, and treat zero as "nobody reads this." The agent learns the policy from the parameter group before the first scan. On ElastiCache, CONFIG GET is blocked, so it comes from describe_cache_parameters on the replication group's parameter group, not from the server.

For a one-off instead of an agent, redis-cli --memkeys -h <reader> does the full-scan version of this and prints the biggest key per type. It walks every key, so it belongs on a replica and off peak.

Step 2: Aggregate by Pattern, Extrapolate, and Rank

Two hundred thousand keys out of thirty million is a 0.7% sample, which is plenty to find every pattern that holds more than about half a percent of memory. The aggregation scales sampled bytes by DBSIZE over the sample count and computes, per pattern, the share with no TTL and the idle-time distribution.

That table already tells the story that the 91% gauge could not. The session cache is two-thirds of the memory, almost none of it expires, and the median key has not been read in nineteen days. The leaderboard is a single key you cannot shard or evict, and every ZRANGE on it blocks the event loop while it runs. The rate-limit keys are fine. The feed lists are a dead cache: a third have a TTL, two thirds do not, which usually means a code path added the TTL in one place and forgot it in another.

Step 3: Read the Eviction Facts Before Anyone Forms an Opinion

Memory pressure shows up as three different symptoms depending on policy, and the agent collects the facts for all three before the model is consulted.

# redismem/facts.py
import boto3, datetime as dt
cw = boto3.client("cloudwatch")

def metric(replication_group: str, name: str, stat: str, days: int = 14) -> list[float]:
    end = dt.datetime.utcnow()
    resp = cw.get_metric_statistics(
        Namespace="AWS/ElastiCache", MetricName=name,
        Dimensions=[{"Name": "ReplicationGroupId", "Value": replication_group}],
        StartTime=end - dt.timedelta(days=days), EndTime=end,
        Period=3600, Statistics=[stat])
    return [d[stat] for d in sorted(resp["Datapoints"], key=lambda d: d["Timestamp"])]

def facts(rg: str) -> dict:
    mem = metric(rg, "DatabaseMemoryUsagePercentage", "Maximum")
    return {
        "mem_pct_p95": sorted(mem)[int(len(mem) * 0.95)],
        "evictions_per_hour_max": max(metric(rg, "Evictions", "Sum")),
        "swap_bytes_max": max(metric(rg, "SwapUsage", "Maximum")),
        "cache_hit_rate_p50": sorted(metric(rg, "CacheHitRate", "Average"))[len(mem) // 2],
        "replication_lag_max_s": max(metric(rg, "ReplicationLag", "Maximum")),
    }

Self-hosted Redis gives the same facts from INFO: used_memory against maxmemory, evicted_keys and expired_keys as counters you difference between runs, keyspace_hits and keyspace_misses for the hit rate, and mem_fragmentation_ratio for the one problem a bigger node will not fix.

The rules the wrapper applies before the model sees anything:

• Evictions with a volatile-* policy and a mostly no-TTL keyspace is a policy failure, not a sizing failure. volatile-lru can only evict keys that have a TTL. When 97% of memory has none, Redis runs out of candidates, and clients start seeing OOM command not allowed when used memory > 'maxmemory' on every write. This is the single most common cause of a "Redis is full" page, and the fix is a TTL, not a node.
• Evictions with allkeys-lru and a hit rate above 95% is healthy. The cache is doing what a cache does. Nothing to fix, and the agent says so rather than inventing work.
• Evictions with allkeys-lru and a hit rate falling below 80% means the working set outgrew the node. This is the only case where right-sizing upward is the correct first answer.
• Fragmentation ratio above 1.5 with memory under 60% means the allocator is holding freed memory. activedefrag yes in the parameter group reclaims it; a bigger node just fragments more slowly.
• Any SwapUsage above a few megabytes is an emergency. ElastiCache reserves 25% of the node for replication and overhead by default via reserved-memory-percent, and swap means something ate into it.

Step 4: The Model Classifies One Pattern at a Time

The model never sees key names beyond the normalized pattern and never sees a value. It gets the pattern's row from the table, the cluster facts, the policy, and a hint about the owner, and returns a strict verdict.

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": [
            "add_ttl", "shorten_ttl", "split_big_key", "change_policy",
            "right_size_node", "healthy", "needs_human"]},
        "ttl_seconds": {"type": ["integer", "null"]},
        "evidence": {"type": "string"},
        "what_breaks_if_wrong": {"type": "string"},
    },
    "required": ["verdict", "evidence", "what_breaks_if_wrong"],
}

SYSTEM = (
    "You review one Redis key pattern from a sampled keyspace report.\n"
    "add_ttl: the pattern has no TTL, its median idle time is far longer than "
    "any plausible read interval, and it is a cache (reconstructable from a "
    "source of truth). Propose a TTL no longer than 2x the p90 idle time of the "
    "keys that ARE being read.\n"
    "split_big_key: a single key over 100 MB or over 1M members. Name the "
    "sharding scheme (by bucket, by time window) and the commands that block.\n"
    "change_policy: evictions occur under a volatile-* policy while most memory "
    "has no TTL. Recommend allkeys-lru only if every pattern is a cache; "
    "otherwise recommend add_ttl on the offenders and keep the policy.\n"
    "right_size_node: only when the policy is allkeys-* and hit rate is "
    "falling. Never recommend a larger node while a no-TTL pattern over 20% "
    "of memory exists.\n"
    "needs_human: the pattern might be a source of truth (queues, locks, "
    "counters, idempotency keys, anything with 'lock', 'job', 'order', 'tx' "
    "in it). Expiring those loses data. Say so and stop.\n"
    "Cite the numbers you were given. Do not infer what the values contain."
)

The ordering in that prompt is the point. A model asked "Redis is at 91%, what should we do" will say "scale up" because that is what the training data says. A model given the table and told it may not recommend a node while a no-TTL pattern dominates memory produces the answer an experienced SRE would: fix the sessions, then look again.

The owner hint comes from two sources. The key prefix is matched against a redis_prefixes annotation on each component in the service catalog, which the Backstage MCP server exposes alongside the on-call rotation. When no component claims a prefix, the agent greps the organization's repositories for the literal prefix string and reports the top repositories as a lead, clearly labelled as a guess.

Step 5: Fixes Ship as Pull Requests, With One Gated Exception

TTLs go into the application, not into Redis. For sess:{hex}, the PR changes the HSET call to HSET followed by EXPIRE, or to a single SET ... EX where the value is a string, and the PR body carries the evidence row so the reviewer sees why 86,400 seconds was chosen. The agent opens it against the repository the owner hint identified, with the same PR discipline as the RDS cost agent: one pattern per PR, title states the memory it frees, body states what breaks if the TTL is wrong.

The backfill is the exception. Shipping the code change only gives new sessions a TTL. The eighteen million existing keys stay forever unless something expires them, and that something is the only write the agent can perform:

# redismem/backfill.py
def backfill_ttl(r_primary, prefix_glob: str, ttl: int, per_second: int, evictions_rising) -> int:
    cursor, done = 0, 0
    while True:
        cursor, keys = r_primary.scan(cursor=cursor, match=prefix_glob, count=500)
        p = r_primary.pipeline(transaction=False)
        for k in keys:
            p.expire(k, ttl, nx=True)       # NX: only keys with no TTL. Redis 7+.
        done += sum(1 for ok in p.execute(raise_on_error=False) if ok is True)
        if cursor == 0 or evictions_rising():
            return done
        time.sleep(len(keys) / per_second)

EXPIRE ... NX is what makes this safe to re-run: a key that already has a TTL is untouched, so an interrupted backfill picks up where it left off without shortening anything a developer set deliberately. The rate is a fifth of the primary's observed write throughput, and the loop stops the moment evictions start rising, because a backfill that gives eleven gigabytes a TTL will, by design, cause a wave of expirations and the agent should not be adding writes while that happens.

Policy changes go through Terraform. When the verdict is change_policy, the PR is a one-parameter diff:

resource "aws_elasticache_parameter_group" "app" {
  name   = "app-redis7"
  family = "redis7"

  parameter {
    name  = "maxmemory-policy"
    value = "allkeys-lru"      # was volatile-lru; 97% of memory had no TTL
  }
  parameter {
    name  = "activedefrag"
    value = "yes"              # frag ratio 1.62 at 58% memory
  }
}

Most parameters apply without a restart on ElastiCache, but maxmemory-policy changes behaviour immediately, so the PR body says what will start being evicted the second it merges.

Right-sizing is a recommendation with a number. When the agent is allowed to say right_size_node, it attaches the node class, the fourteen-day memory p95, and the on-demand monthly delta from the pricing API. The reverse case is the one that pays: after the session TTL lands and memory settles at 38%, the next run recommends stepping down a node class, with the same evidence. On a three-node cache.r7g.xlarge replication group that is roughly half the monthly bill, and it was never a capacity problem to begin with. The caching patterns guide covers why the cache exists; this agent covers why it is three times larger than it should be.

Guardrails That Live in Code

The prompt expresses judgement. These are enforced by the wrapper and by Redis itself.

• A dedicated ACL user with an allowlist, not a denylist. On Redis 6+ or ElastiCache RBAC: on ~* -@all +scan +type +ttl +pttl +dbsize +info +memory|usage +object|idletime +object|freq. KEYS, DEL, UNLINK, FLUSHALL, FLUSHDB, CONFIG, MONITOR, and DEBUG are not available to this user even if a prompt injection in a key name asks for them. The backfill uses a second user that adds only +expire, and that user's credentials are released by the approval step, not held by the agent.
• Reader endpoint for every read. The sampler refuses to connect to a host whose INFO replication reports role:master.
• Approval before the only write. The plan is one Slack message following the approval-gate pattern: pattern, estimated keys, proposed TTL, rate, estimated duration, and what the evictions graph will do. One approval covers one pattern on one cluster.
• Action budget and auto-demotion. One backfill per cluster per day, a cap on keys expired per run, and any eviction spike or p99 latency climb during the backfill demotes the agent to read-only for 24 hours, as in the circuit-breaker post.
• Source-of-truth patterns are untouchable. Any pattern the catalog marks durable: true, and any pattern matching the lock, job, order, or idempotency heuristics, is excluded from add_ttl by the wrapper regardless of the model's verdict. The model may suggest it; the code drops it.

Run it in shadow mode first: the agent posts the table and the verdicts, a human does the backfill by hand, and the team scores whether the pattern the agent blamed was the pattern that mattered. The number to watch is needs_human recall. An unnecessary needs_human costs ten minutes of reading; an add_ttl on a pattern that turned out to be a job queue costs the jobs.

Honest Limits

Sampling finds patterns, not needles. A single 400 MB key among thirty million has about a one-in-150 chance of landing in a 0.7% sample. The agent catches it anyway most of the time, because big keys show up as outliers in MEMORY USAGE the moment they are sampled, but a scheduled full --memkeys pass on a replica once a week is the honest complement.

Idle time is a proxy for reads, not a measurement. Redis has no per-prefix hit rate. A pattern with a 19-day median idle time is almost certainly dead, but a pattern with a two-hour median might be read once an hour by a batch job that matters. The verdicts say "likely" where the data only supports likely, and the needs_human heuristics exist because prefix names lie.

Cluster mode multiplies the work. SCAN is per node, so the sampler walks every shard's replica, and the interesting finding is often skew: one shard at 88% and the rest at 40% because a hash tag put every tenant's keys on the same slot. The agent reports per shard and flags skew above 20%, but the fix is a key-design change, and that is a design review, not a PR the agent can write.

Finally, the agent cannot see what a value is for. It knows sess:{hex} is a hash with no TTL that nobody reads. It does not know whether the login flow treats a missing session as "log in again" or as a crash. The PR it opens makes that question unavoidable for a human who can answer it, which is the correct amount of autonomy for an agent whose one write can log out every user you have.