Build an ECR Cost Agent: Expire Untagged and Never-Pulled Images, Generate Lifecycle Policies, and Prove No Cluster Still Runs What You Delete

What This Agent Does

An ECR cost agent inventories every private repository in the account, finds the images nothing has pulled in months, proves none of them is referenced by a running pod, a rollback target, a Lambda function, or an active ECS task definition, and then writes a lifecycle policy per repository that matches the team's actual tagging convention. The inventory and the in-use proof are deterministic boto3 and kubectl calls. The LLM does one thing: it reads a sample of a repository's tags and proposes the tagPatternList rules a human would have written, which the agent then runs through ECR's own lifecycle preview before anything expires. The only write path is a pull request to the Terraform that owns the repositories. The agent never calls batch-delete-image.

The storage line itself is modest at $0.10 per GB-month, so a 2 TB registry is only about $200 a month. What makes stale images expensive is everything attached to them: Inspector enhanced scanning keeps rescanning every image inside its rescan window whenever a new CVE lands, cross-region replication copies every push to every destination, and the CVE triage queue fills with findings on images that last ran in 2024. Deleting what nobody uses is a cost fix, a security fix, and a signal-to-noise fix in one PR.

Where ECR Spend and Risk Actually Hide

The second row is the big one in every account I have audited. A monorepo with 40 services and 30 merges a day pushes roughly 1,200 images a day. Each one is a 400 MB tag that will be pulled exactly once by the CI job that built it, or never.

Step 1: Collect Repositories, Images, and Pull Times

The collector needs ecr:DescribeRepositories, ecr:DescribeImages, ecr:GetLifecyclePolicy, ecr:DescribeRegistry, and ecr:DescribePullThroughCacheRules. Nothing else. The one field that makes the whole thing work is lastRecordedPullTime, which ECR has tracked per image since 2022. It is updated with roughly 24-hour granularity, which is fine for a 90-day idle test.

# collect_ecr.py — read-only inventory
import boto3
from datetime import datetime, timedelta, timezone

ecr = boto3.client("ecr")
NOW = datetime.now(timezone.utc)

def repos():
    for page in ecr.get_paginator("describe_repositories").paginate():
        for r in page["repositories"]:
            try:
                policy = ecr.get_lifecycle_policy(repositoryName=r["repositoryName"])["lifecyclePolicyText"]
            except ecr.exceptions.LifecyclePolicyNotFoundException:
                policy = None
            yield {"name": r["repositoryName"], "uri": r["repositoryUri"],
                   "scan_on_push": r["imageScanningConfiguration"]["scanOnPush"],
                   "lifecycle_policy": policy}

def images(repo):
    for page in ecr.get_paginator("describe_images").paginate(repositoryName=repo):
        for i in page["imageDetails"]:
            pulled = i.get("lastRecordedPullTime")
            yield {
                "digest": i["imageDigest"],
                "tags": i.get("imageTags", []),
                "size_mb": round(i["imageSizeInBytes"] / 1e6),
                "pushed_days": (NOW - i["imagePushedAt"]).days,
                "pulled_days": (NOW - pulled).days if pulled else None,
                "media_type": i.get("imageManifestMediaType", ""),
            }

Two things about the numbers this produces:

• Do not sum imageSizeInBytes and call it the bill. Layers shared between images in the same repository are stored once. The per-image size ranks candidates; the real monthly figure comes from Cost Explorer filtered to the ECR service and the TimedStorage-ByteHrs usage type. Put both in the report and label which is which.
• Multi-arch images lie about being untagged. A tagged image index points at two or more platform manifests that describe-images lists as untagged. A naive "expire all untagged" deletes the arm64 half of a tagged release. The collector marks any digest whose media_type is an OCI index or Docker manifest list, then calls batch_get_image on it and records the child digests as protected. Whatever ECR's lifecycle engine does with indexes in your registry today, the preview check in Step 4 must confirm that no child of a tagged index appears in the expiring set.

Replication and pull-through state come from describe_registry and describe_pull_through_cache_rules, one call each, and go into the same report.

Step 2: Build the In-Use Set From Every Consumer

This is the step that separates a safe deletion from a Friday-evening ImagePullBackOff. The agent collects every image reference that could legitimately be pulled again and resolves each to a digest. Tags are not enough, because a tag can be moved; the running pod's imageID is the digest that actually started.

# Running containers, by digest, every cluster in the kubeconfig
for ctx in $(kubectl config get-contexts -o name); do
  kubectl --context "$ctx" get pods -A \
    -o jsonpath='{range .items[*]}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}'
done | grep 'dkr.ecr' | sed 's/.*@//' | sort -u > inuse_running.txt

# Rollback targets: images in every ReplicaSet still retained by a Deployment
kubectl get rs -A -o jsonpath='{range .items[*]}{range .spec.template.spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | grep 'dkr.ecr' | sort -u > inuse_rollback_tags.txt

ReplicaSets give you tags, not digests, so the agent resolves each tag with describe_images --image-ids imageTag=... at collection time. The default revisionHistoryLimit is 10, which means a Deployment can roll back to any of its last ten images; the lifecycle rule for that repository must keep at least that many, and the agent reads the actual limit per Deployment rather than assuming the default.

The other consumers are easy to forget and each has its own API:

• Helm release history. helm history per release lists revisions that helm rollback can target. The agent renders the retained revisions' manifests and extracts image references. Default --history-max is 10.
• Argo CD. Application history in the status.history field of each Application resource records the Git revision, not the image, so the agent resolves images by reading the manifests at those revisions. The GitOps pattern for agents already gives it read access to the repo.
• Lambda container images. list_functions filtered to PackageType == Image, then get_function for the Code.ResolvedImageUri, which is already a digest.
• ECS. Every ACTIVE task definition revision, since a service can be updated to any of them. list_task_definitions --status ACTIVE then describe_task_definition for container images.
• Images referenced by a Kyverno policy or a signed-image allowlist. If you enforce signatures the way the Cosign and Kyverno post describes, the policy may name specific digests. Those are protected too.

The union of all of this is the in-use set. Everything in it, plus everything a retained index references, is off-limits regardless of age or pull time. The agent stores it as a plain file next to the run's audit record, so a reviewer can grep it.

Step 3: Classify Deterministically, Then Let the Model Write Rules

Classification is a lookup, not a prompt:

def classify(img, inuse, protected_children):
    if img["digest"] in inuse or img["digest"] in protected_children:
        return "protected"
    if not img["tags"]:
        return "untagged" if img["pushed_days"] > 7 else "fresh"
    if img["pulled_days"] is None and img["pushed_days"] > 30:
        return "never_pulled"
    if img["pulled_days"] is not None and img["pulled_days"] > 90:
        return "stale"
    return "active"

Where the model earns its tokens is the lifecycle policy. ECR lifecycle rules match on tag status and tag patterns, so the right rules depend on how a team tags. One repository uses sha-abc1234 for CI and v1.42.0 for releases; another uses main-20261016-abc1234; a third tags with Jira ticket numbers. A regex on 200 sample tags gets this wrong in surprising ways. A model does not, provided it is only allowed to fill in a fixed schema and its output is previewed before use.

You are proposing an ECR lifecycle policy for one repository. Return JSON only,
matching the ECR lifecycle policy schema exactly.

Input: 200 sample tags, the count of images per classification
(untagged, never_pulled, stale, active, protected), the maximum Deployment
revisionHistoryLimit and Helm history-max that reference this repo, and the
newest protected image's position when images are sorted by push date.

Rules:
- Rule 1 must expire untagged images older than 7 days.
- Identify the CI/commit tag pattern and the release tag pattern from the
  sample. Express each as a tagPatternList glob (ECR supports * wildcards only).
- The count rule for the CI pattern must keep at least
  max(revisionHistoryLimit, helm_history_max, protected_position) + 5 images.
- Release-pattern images expire by age only, never below 180 days.
- Never emit a rule with tagStatus "any".
- Add a "reasoning" field, two sentences, quoting the tag examples you
  matched. This field is stripped before the policy is applied.

A typical result for the monorepo case:

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Expire untagged manifests after 7 days",
      "selection": { "tagStatus": "untagged", "countType": "sinceImagePushed",
                     "countUnit": "days", "countNumber": 7 },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 10,
      "description": "Keep the last 40 CI builds (sha-*)",
      "selection": { "tagStatus": "tagged", "tagPatternList": ["sha-*"],
                     "countType": "imageCountMoreThan", "countNumber": 40 },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 20,
      "description": "Expire release images (v*) after 365 days",
      "selection": { "tagStatus": "tagged", "tagPatternList": ["v*"],
                     "countType": "sinceImagePushed", "countUnit": "days",
                     "countNumber": 365 },
      "action": { "type": "expire" }
    }
  ]
}

Two ECR semantics the prompt has to encode because the model will otherwise get them wrong. First, an image with several tags is matched if any one of its tags matches a rule, so a "keep" tag does not protect an image from a rule matching its other tag. That is why protection is done by sizing the count rule, not by adding tags. Second, rules are evaluated by priority and an image is only ever matched by the first rule that selects it, so the untagged rule goes first and the broad patterns go last.

Step 4: Preview, Diff Against the In-Use Set, Then Open the PR

ECR will tell you exactly what a policy would expire before you apply it. This is the guardrail, and it is free.

aws ecr start-lifecycle-policy-preview \
  --repository-name platform/orders-api \
  --lifecycle-policy-text file://proposed.json

# poll until status is COMPLETE, then
aws ecr get-lifecycle-policy-preview \
  --repository-name platform/orders-api \
  --query 'previewResults[?action.type==`EXPIRE`].imageDigest' --output text \
  | tr '\t' '\n' | sort > would_expire.txt

comm -12 would_expire.txt inuse_all.txt

If comm prints anything, the run fails for that repository and the finding goes to a human with the offending digests and the workload that references each one. No retry loop, no "raise the count and try again" by the model; a policy that would have deleted a rollback target is a policy whose assumptions were wrong, and a person should look at why.

If the preview is clean, the agent writes the policy into the Terraform module that owns the repository and opens a PR through the same approval gate as the other cost agents:

resource "aws_ecr_lifecycle_policy" "orders_api" {
  repository = aws_ecr_repository.orders_api.name
  policy     = file("${path.module}/lifecycle/orders-api.json")
}

The PR body carries the preview summary: images that would expire, GB reclaimed per the per-image sizes, the Cost Explorer figure for context, the count of Inspector findings that disappear with them, and the reasoning field from the model so the reviewer can check the tag-pattern inference against their own convention. It also names the revisionHistoryLimit the count rule was sized from, because that is the question every reviewer asks first.

Two companion changes go into the same PR when the collector finds them missing:

• A pull-through cache rule for Docker Hub and other upstreams, so base images are fetched once per region instead of once per node through the NAT gateway. The NAT gateway cost post shows how much of that egress is image pulls; this is where it gets fixed. Pair it with a repository creation template so cached repositories inherit a lifecycle policy on creation.
• Replication filters. If registry replication is configured with no repository filters, every CI tag is being copied to every destination region. The agent proposes filters on the release prefix only.

aws ecr create-pull-through-cache-rule \
  --ecr-repository-prefix docker-hub \
  --upstream-registry-url registry-1.docker.io \
  --credential-arn arn:aws:secretsmanager:eu-west-1:123456789012:secret:ecr-pullthroughcache/docker-hub

The secret name must start with ecr-pullthroughcache/, or the call is rejected. Worth stating in the PR, since it is the one thing people retry three times.

What a First Run Looks Like

On a 140-repository account the inventory took about six minutes, almost all of it in describe-images pagination, and the in-use collection across four EKS clusters, 31 Lambda functions, and two ECS clusters took another two. The model call per repository was a few thousand input tokens, so the whole classification pass cost well under a dollar.

Eighty-four of the 140 repositories had no lifecycle policy at all. Twelve had one that only expired untagged images, which is the policy everyone copies from the docs and which does nothing about the never-pulled bucket. The preview check failed on three repositories, all for the same reason: a Deployment with revisionHistoryLimit: 50 set during an incident years ago, which the count rule had been sized against correctly but a Helm release in another namespace referenced an older tag the collector only found on the second pass. The fix was to run the Helm collector per cluster rather than per kubeconfig context default, which is the kind of bug a deterministic preview catches and a confident model summary would have hidden.

The three preview failures were also the strongest argument for the design. Nobody on the platform team knew those images were still rollback targets, and a policy applied by hand would have expired them.

Limits Worth Stating in the PR Template

• The in-use set is a snapshot. A pod that starts tomorrow from a tag that expired today will fail to pull. The count rules are sized with headroom, and any repository where the gap between the count rule and the protected position is under five images is routed to a human instead of a PR.
• lastRecordedPullTime is per registry, not per consumer. It cannot tell you which cluster pulled an image, and it has roughly a day of lag. The 90-day stale threshold absorbs the lag. Do not lower it to a week.
• Replicated repositories report their own pull times. A release image pulled only in the replica region looks never-pulled in the source. The collector runs in every region with a registry, and an image is only never_pulled if that is true everywhere.
• It does not shrink images. A 400 MB image that should be 60 MB is a multi-stage build problem, and the per-image size table in the report is where those repositories show up. This agent reclaims storage; it does not reduce what each push costs.
• Run it monthly, diff against last month. The second run should show the untagged and never-pulled buckets near zero on every repository with a policy, and anything new in the no-policy list is a repository created outside Terraform, which is its own finding.