Build an AWS Tag Compliance Agent: Find Untagged Resources, Infer the Owner from CloudTrail and Terraform State, and Backfill Cost Allocation Tags via PR

What This Agent Does

A tag compliance agent finds every AWS resource that is missing the tags your cost allocation depends on (team, cost-center, service), works out who actually owns each one from evidence rather than guesswork, and opens a pull request that adds the tags in Terraform. The evidence is CloudTrail (who created it), your Terraform state bucket (which repo manages it, and therefore which CODEOWNERS team), and the parent resource (the instance a volume is attached to, the cluster an ENI belongs to). The LLM gets that evidence pre-chewed and decides between "owner is clear, open the PR", "two teams could own this, ask both", and "this was created by a service, fix the propagation setting instead". Its IAM role explicitly denies every tagging action, so a tag only lands when a human merges.

It is the missing precondition for every other cost agent on this site. The cost anomaly agent cannot explain a spike that lands in the (untagged) bucket, and the waste reclamation agent cannot tell a forgotten volume from a disaster-recovery copy when neither carries an owner. The Kubernetes sibling is cost allocation per namespace; this is the same showback problem one layer down, where the unit of ownership is an ARN instead of a namespace.

Why "List Untagged Resources" Is Harder Than It Sounds

The obvious tool is the Resource Groups Tagging API, and it is the wrong one for this job:

The agent uses Config for the inventory and Cost Explorer for the money. Config answers "which ARNs lack the tag", Cost Explorer answers "how much that is costing you", and the join between them is what makes the finding list rank by dollars instead of by count.

Step 1: A Role That Reads Everything and Tags Nothing

The explicit Deny is the guardrail. It outlives any Allow somebody adds later, and it is the reason the agent cannot "just fix it" when a prompt tells it to.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadInventoryAndEvidence",
      "Effect": "Allow",
      "Action": [
        "config:SelectAggregateResourceConfig", "config:SelectResourceConfig",
        "ce:GetCostAndUsage", "ce:GetCostAndUsageWithResources",
        "ce:ListCostAllocationTags",
        "cloudtrail:LookupEvents",
        "athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults",
        "s3:GetObject", "s3:ListBucket",
        "ec2:DescribeInstances", "ec2:DescribeVolumes", "ec2:DescribeNetworkInterfaces",
        "autoscaling:DescribeAutoScalingGroups",
        "organizations:ListTagsForResource", "organizations:DescribeAccount"
      ],
      "Resource": "*"
    },
    {
      "Sid": "NeverTag",
      "Effect": "Deny",
      "Action": [
        "tag:TagResources", "tag:UntagResources",
        "ec2:CreateTags", "ec2:DeleteTags",
        "rds:AddTagsToResource", "rds:RemoveTagsFromResource",
        "s3:PutBucketTagging", "lambda:TagResource", "dynamodb:TagResource",
        "elasticloadbalancing:AddTags", "eks:TagResource",
        "ce:UpdateCostAllocationTagsStatus", "ce:StartCostAllocationTagBackfill"
      ],
      "Resource": "*"
    }
  ]
}

The s3:GetObject grant is scoped in the real policy to the Terraform state bucket only. Verify the boundary before trusting it:

aws ec2 create-tags --resources i-0abc123def456 --tags Key=team,Value=probe
# An error occurred (UnauthorizedOperation) ... explicit deny in an identity-based policy

Why not let the agent tag directly, given that tagging is harmless? Because out-of-band tags on a Terraform-managed resource are drift. The next terraform apply either removes them or, with ignore_tags configured, silently leaves two sources of truth. A tag is a claim of ownership, and the team being named should see it in a diff before it becomes a line on their invoice.

Step 2: Inventory From AWS Config, Money From Cost Explorer

One query against the organization aggregator covers every account and region. The required-tag check happens client-side, because the resource has to be in the result set before you can notice what it lacks.

# inventory.py — every resource missing a required tag, with its 14-day cost
import json, boto3
from collections import defaultdict

REQUIRED = {"team", "cost-center", "service"}
TYPES = ["AWS::EC2::Instance", "AWS::EC2::Volume", "AWS::RDS::DBInstance",
         "AWS::RDS::DBCluster", "AWS::S3::Bucket", "AWS::Lambda::Function",
         "AWS::DynamoDB::Table", "AWS::ElasticLoadBalancingV2::LoadBalancer",
         "AWS::EKS::Cluster", "AWS::ECS::Service", "AWS::ElastiCache::CacheCluster",
         "AWS::EC2::NatGateway", "AWS::EC2::NetworkInterface"]

cfg = boto3.client("config")
ce = boto3.client("ce")

def untagged(aggregator="org-aggregator"):
    expr = ("SELECT resourceId, resourceType, accountId, awsRegion, arn, tags, "
            "resourceCreationTime WHERE resourceType IN ('" + "','".join(TYPES) + "')")
    token = None
    while True:
        kw = {"ConfigurationAggregatorName": aggregator, "Expression": expr, "Limit": 100}
        if token:
            kw["NextToken"] = token
        page = cfg.select_aggregate_resource_config(**kw)
        for raw in page["Results"]:
            r = json.loads(raw)
            present = {t["key"] for t in r.get("tags", [])}
            missing = REQUIRED - present
            if missing:
                yield {**r, "missing": sorted(missing), "present_tags": r.get("tags", [])}
        token = page.get("NextToken")
        if not token:
            return

def resource_cost_14d():
    """Resource-level cost. Only 14 days are available, but that is enough to rank."""
    import datetime as dt
    end = dt.date.today(); start = end - dt.timedelta(days=14)
    cost = defaultdict(float)
    kw = {"TimePeriod": {"Start": start.isoformat(), "End": end.isoformat()},
          "Granularity": "DAILY", "Metrics": ["UnblendedCost"],
          "Filter": {"Tags": {"Key": "team", "MatchOptions": ["ABSENT"]}},
          "GroupBy": [{"Type": "DIMENSION", "Key": "RESOURCE_ID"}]}
    while True:
        page = ce.get_cost_and_usage_with_resources(**kw)
        for day in page["ResultsByTime"]:
            for g in day["Groups"]:
                cost[g["Keys"][0]] += float(g["Metrics"]["UnblendedCost"]["Amount"])
        if "NextPageToken" not in page:
            return cost
        kw["NextPageToken"] = page["NextPageToken"]

Two details that matter. The ABSENT match option is how Cost Explorer expresses "no value for this tag key", and it only works after the key has been activated as a cost allocation tag; if ce list-cost-allocation-tags --status Inactive still lists team, the agent's first finding is that activation itself, because nothing downstream can work until it is on. And resource-level cost covers 14 days, so the ranking is a recent-spend ranking; the monthly coverage number in the last section comes from the plain get-cost-and-usage call instead.

Step 3: Infer the Owner From Three Independent Sources

The point of the agent is not the list. Any script produces the list. The point is attaching a defensible owner to each line, and that comes from signals that do not share a failure mode.

Signal 1: Which Terraform State File Contains the ARN

Nightly, the agent indexes every state file in the state bucket and maps each ARN to the key it was found in. The key encodes repo and directory by convention (payments/ledger-api/prod/terraform.tfstate), and the repo's CODEOWNERS names a team.

# state_index.py — ARN -> (state key, codeowners team), rebuilt nightly
import json, boto3
s3 = boto3.client("s3")

def index_state(bucket):
    arn_to_key = {}
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket):
        for obj in page.get("Contents", []):
            if not obj["Key"].endswith(".tfstate"):
                continue
            state = json.loads(s3.get_object(Bucket=bucket, Key=obj["Key"])["Body"].read())
            for res in state.get("resources", []):
                for inst in res.get("instances", []):
                    arn = inst.get("attributes", {}).get("arn")
                    if arn:
                        arn_to_key[arn] = obj["Key"]
    return arn_to_key

This is the strongest signal, because it also tells you where the fix goes. It is not proof of ownership on its own: a shared platform/networking state owns NAT gateways that six teams use, and the correct tag there is team=platform with a separate chargeback split, which is a policy decision and not an inference.

Signal 2: Who Created It, According to CloudTrail

For resources under 90 days old, lookup-events by resource name returns the creation call. Older than that, the same question goes to Athena over the CloudTrail bucket.

def creator(resource_id, region):
    ct = boto3.client("cloudtrail", region_name=region)
    events = ct.lookup_events(LookupAttributes=[
        {"AttributeKey": "ResourceName", "AttributeValue": resource_id}], MaxResults=50)["Events"]
    create = [e for e in events if e["EventName"].startswith(("Run", "Create"))]
    if not create:
        return None
    ident = json.loads(create[-1]["CloudTrailEvent"])["userIdentity"]
    arn = ident.get("arn", "")
    role = arn.split("assumed-role/")[-1].split("/")[0] if "assumed-role/" in arn else ident.get("type")
    session = arn.rsplit("/", 1)[-1]
    return {"role": role, "session": session, "time": create[-1]["EventTime"].isoformat()}

With IAM Identity Center, the session name is the engineer's username, and a username maps to a team through your directory. With a CI role, the session name is a pipeline run, and the signal says nothing about ownership. It says "look at the pipeline", which almost always leads back to Signal 1. The prompt in Step 4 encodes that rule so the model never writes "owned by github-actions-deploy".

Signal 3: The Parent Resource

An untagged EBS volume attached to a tagged instance, an ENI on a tagged EKS cluster, an instance launched by a tagged Auto Scaling group: these are not ownership mysteries, they are propagation bugs. The agent resolves the parent with the obvious Describe call and copies its tags as the proposed value, but the finding type changes, because tagging the child by hand fixes one resource and the next scale-out creates ten more untagged ones.

Step 4: What the Model Decides

One JSON object per resource: the missing keys, the 14-day cost, the state-file hit, the creator, the parent's tags, and the account's own tags from Organizations. The model classifies and writes prose. It does not choose tag values outside the allowed list.

You are assigning cost allocation tags to untagged AWS resources. Allowed values for
team and cost-center are in the attached tag policy; never invent one. Output one verdict:
  owner_confident  — at least two independent signals name the same team (state file
                     CODEOWNERS, human CloudTrail creator, parent resource tag, or a
                     single-team account). Propose the tags. Severity by 14-day cost.
  owner_split      — signals name different teams. Propose nothing. Draft one question to
                     each team quoting the signal that pointed at them.
  fix_propagation  — the resource has a parent whose tags it should have inherited.
                     Propose the change on the parent (ASG, launch template, EC2NodeClass,
                     CSI driver), not on the child. List the sibling count.
  orphan_candidate — creator's account is disabled in the directory AND no state file
                     contains the ARN. Hand off to the waste reclamation queue. Do not tag.
  hold             — anything else. Say exactly which signal would resolve it.
Rules:
- A CloudTrail creator that is a CI or Terraform role is NOT an owner signal. Count it as zero.
- A shared state file (platform/, networking/, shared-services/) is a platform-owned
  resource. Propose team=platform and flag it for chargeback split; do not guess a tenant.
- Tags name a team, never a person. If the only signal is a username, output hold and
  name the directory lookup that is missing.
- Never propose removing or changing an existing tag value, even one that looks wrong.

A finding comes back shaped for the person who will approve it:

{
  "arn": "arn:aws:rds:eu-west-1:123456789012:db:ledger-replica-2",
  "missing": ["cost-center", "team"],
  "cost_14d_usd": 412.80,
  "verdict": "owner_confident",
  "evidence": "State file payments/ledger-api/prod/terraform.tfstate (CODEOWNERS: @org/payments). CloudTrail CreateDBInstance 2026-06-03 by AWSReservedSSO_PowerUser/r.okafor, payments directory group. Account 123456789012 is multi-team; no account signal.",
  "proposed_tags": {"team": "payments", "cost-center": "CC-4410", "service": "ledger-api"},
  "fix": "PR #522 adds tags to aws_db_instance.ledger_replica_2; provider default_tags change included so the rest of the module inherits.",
  "confidence": 0.92
}

Confidence under 0.7 is written into the PR body as a note, never as a diff. The threshold is the same one the secret leak triage agent uses: a wrong ownership tag is a wrong invoice, and the first team charged for someone else's database stops trusting every number the agent produces.

Step 5: The Fix Is a Terraform PR That Can Only Touch Tags

Per the pull requests, not API calls rule, the agent writes to a branch in the repo the state index pointed at. The preferred diff is at the provider, because default_tags fixes every resource in the module at once, which is also the cheapest of the Terraform cost patterns:

# Before
provider "aws" {
  region = "eu-west-1"
}

# After: every taggable resource in this root module inherits these
provider "aws" {
  region = "eu-west-1"
  default_tags {
    tags = {
      team        = "payments"
      cost-center = "CC-4410"
      service     = "ledger-api"
      managed-by  = "terraform"
    }
  }
}

Resources that need a value different from the module default get an explicit tags block, which merges over default_tags. The plan gate then enforces the only invariant that matters: every planned change is an in-place update, and the before and after differ in nothing except tags.

terraform show -json tfplan | jq -e '
  [.resource_changes[]
    | select(.change.actions != ["no-op"])
    | select(
        (.change.actions == ["update"]
         and ((.change.before | del(.tags, .tags_all)) == (.change.after | del(.tags, .tags_all)))
        ) | not)
  ] | length == 0' >/dev/null || { echo "plan changes something other than tags; aborting"; exit 1; }

A replacement, a forced new resource because a tag change on some older resource types triggers one, or any attribute drift that happened to be sitting in the module all fail this gate, and the PR does not open. Feed the same allowlist to the Terraform plan review agent if you already run it, so there is one place that knows what "tags only" means.

Resources not in any state file get an issue with an import block proposal and the evidence attached, not a PR. fix_propagation findings get a PR against the parent, with the number of currently untagged children in the title, because "tags 1 launch template" gets ignored and "tags 1 launch template, fixes 38 volumes" gets merged.

Stop the Bleeding at Create Time

The backlog shrinks only if the inflow stops. An SCP that denies creation without the required key costs nothing and catches the console and the CLI equally:

{
  "Sid": "DenyUntaggedCompute",
  "Effect": "Deny",
  "Action": ["ec2:RunInstances", "rds:CreateDBInstance", "rds:CreateDBCluster"],
  "Resource": ["arn:aws:ec2:*:*:instance/*", "arn:aws:ec2:*:*:volume/*", "arn:aws:rds:*:*:db:*", "arn:aws:rds:*:*:cluster:*"],
  "Condition": {"Null": {"aws:RequestTag/team": "true"}}
}

Pair it with an Organizations tag policy that pins the allowed values, so Team, team, and teams cannot coexist. Roll the SCP out per OU after the agent has cleared that OU's backlog, not before, or the first thing it blocks is the Auto Scaling group that was about to replace a failed node.

After Merge: Coverage Is the Only Metric

Cost Explorer lags about a day, so the agent checks each merged PR two days later: the resource's spend should have moved out of the team$ group. If it has not, either the tag did not apply (check tags_all in the state) or the key was never activated. Then it runs the one number leadership cares about:

def coverage(month_start, month_end):
    out = ce.get_cost_and_usage(TimePeriod={"Start": month_start, "End": month_end},
        Granularity="MONTHLY", Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "TAG", "Key": "team"}])["ResultsByTime"][0]["Groups"]
    total = sum(float(g["Metrics"]["UnblendedCost"]["Amount"]) for g in out)
    untagged = sum(float(g["Metrics"]["UnblendedCost"]["Amount"]) for g in out if g["Keys"] == ["team$"])
    return 1 - untagged / total, untagged

Graph coverage weekly and alert on the untagged dollar amount growing, not on the percentage falling, for the same reason the namespace showback post alerts on growth: a new region or a migration can move the ratio without anyone doing anything wrong, but untagged spend going up means a new source of untagged resources appeared. Historical months stay untagged in Cost Explorer unless you run ce start-cost-allocation-tag-backfill, which rewrites up to twelve months of billing data with the tags resources carry now. That is one of the actions the agent's role denies, and it should stay a human's button, because it changes last quarter's numbers.

Where It Will Be Wrong

Run it in shadow mode for a few weeks and expect these:

• Config does not record everything. Some resource types are not recordable at all, and some accounts have recording switched off for cost reasons. The agent must report "not in inventory" as a gap in its own coverage, not as zero untagged resources.
• The state index lies after a terraform state mv. The nightly rebuild catches it, but a finding generated between the move and the rebuild points at the old key. Attach the index timestamp to every finding.
• Account-level ownership is a trap in shared accounts. A single-team account is a strong signal. A "prod" account used by twelve teams is no signal, and the model will happily treat it as one unless the account's own tags say shared=true.
• Tag changes are not always free. Changing tags on some resources (older Redshift and certain ElastiCache configurations, among others) forces a replacement in older provider versions. The plan gate catches it, but you need to know why a PR silently did not open.
• Cost Explorer's resource IDs do not match every ARN. EC2 reports instance IDs, S3 reports bucket names, and some services report nothing useful at resource granularity. Join on the identifier Config exposes for that type, and accept that some findings rank by type-level cost instead.
• Directory lookups go stale. A creator who changed teams six months ago still shows up under their old group. Weight the state-file signal above the human-creator signal when they disagree, and let owner_split ask rather than assert.
• The SCP will block something legitimate on day one. Usually an older pipeline that sets tags in a second API call after create. Fix the pipeline, do not loosen the policy.

Takeaway

Untagged spend is not a tagging problem, it is an ownership-evidence problem that happens to surface in the bill. The deterministic part, inventory from Config, dollars from Cost Explorer, creator from CloudTrail, manager from the state bucket, parent from a Describe call, is the ninety percent and it is plain code. The model's job is the narrow one it is good at: look at three signals, decide whether they agree, refuse to guess when they do not, and write the PR body the named team will accept without a meeting. Keep the role's explicit Deny, let the plan gate enforce "tags only", stop the inflow with an SCP after the backlog is clear, and measure coverage in dollars. Every other cost agent you build gets more accurate the week this one ships.